RakeshRamini changes to contentflow v0.1.2#59
Open
RakeshRamini wants to merge 32 commits into
Open
Conversation
|
@RakeshRamini please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
…/new-executor-document-validation-fix
…sets - Add postprovision hook to azure.yaml to run CU setup after infra provisioning - Add seed/analyzers/details_extractor_documents_new1.json for automatic custom analyzer creation via CU REST API - Add seed/pipelines/draft-fetched-details-pipeline.yaml with env var placeholders - Extend post-provision.sh with CU model defaults binding and analyzer auto-discovery - Add text-embedding-3-large model deployment to ai-foundry.bicep
…vate networking - Fix container app ingress port: use targetPort instead of hardcoded port 80 - Fix container app ingress scope: set externalIngress to true (VNet-scoped on internal CAE) - Add pipeline executor env vars to API and Worker container apps - Add queue private DNS zone auto-creation for AILZ mode (new module: queue-dns-zone.bicep) - Fix Cosmos DB default location from eastus2 to eastus - Fix seed pipeline blob prefix mismatch: inputs/ -> input/
… pipeline auto-seeding - Add POST /api/ingest endpoint for multipart document submission - Add IngestPayload/IngestResponse models, IngestService, ingest router - Fix document_validation_executor: flat ProvidedDetails format support, rules_base_path direct settings read, parent dir extraction from content.id.path, cleanup moved to process_input to prevent premature file deletion - Fix ProvidedDetails field names to PascalCase matching rules.json/FetchedDetails - Propagate parent content data in azure_blob_input_discovery to child items - Add rules_base_path, cleanup_input_after_results, cleanup_preserve_files settings to executor_catalog.yaml (version bump 1.2 -> 1.3) - Add pipeline auto-seeding from seed/pipelines/ on startup - Add test_ingest_api.py for integration testing - Add temp-venv to .gitignore
…ing validation results
…-validation-fix feat: Add document validation executor, multipart ingest API, and AILZ infra fixes
…pe category in results json.
…and skip validation for cae dns
Comment out (not remove) all worker-related infrastructure: - Worker container app module and params in main.bicep - Worker App Config keys and queue App Config keys - WORKER_ENGINE_API_ENDPOINT env var from API container app - Worker service definition in azure.yaml - Queue creation in post-provision.sh - Worker endpoint output in post-deploy.sh All sections marked with 'WORKER DISABLED' for easy re-enablement. API executes pipelines directly via background asyncio tasks.
Parse executors and edges from seed YAML into the nodes/edges arrays that the ReactFlow canvas uses for rendering. Previously these were hardcoded to empty arrays, causing blank canvas and empty YAML tab on seeded pipelines.
…cess - Replace Azure AI User (read-only) with Azure AI Developer (read+write) so PATCH /defaults and PUT /analyzers succeed without HTTP 401 - Remove user role assignment from ai-foundry.bicep to avoid RoleAssignmentExists conflict when role was previously assigned via CLI with a different GUID - Switch from --assignee to --assignee-object-id to avoid Microsoft Graph lookup - Add idempotent RoleAssignmentExists detection for clean re-run output - Add sleep 90 for RBAC propagation before CU API calls - Reduce text-embedding-3-large capacity to 40 (quota)
…cess - Replace Azure AI User (read-only) with Azure AI Developer (read+write) so PATCH /defaults and PUT /analyzers succeed without HTTP 401 - Remove user role assignment from ai-foundry.bicep to avoid RoleAssignmentExists conflict when role was previously assigned via CLI with a different GUID - Switch from --assignee to --assignee-object-id to avoid Microsoft Graph lookup - Add idempotent RoleAssignmentExists detection for clean re-run output - Add sleep 90 for RBAC propagation before CU API calls - Reduce text-embedding-3-large capacity to 40 (quota)
- Delete contentflow-worker/ directory (18 files, ~3400 lines) - Delete infra/bicep/main.json (stale 51K-line compiled ARM template) - Delete infra/scripts/post-deploy-worker.sh - Remove worker service definition from azure.yaml - Remove worker-related settings from API (settings.py, dependencies.py) - Remove worker health checks from health_service.py - Remove worker status indicator from Footer.tsx - Remove worker field from apiTypes.ts HealthCheck interface - Remove worker queue creation from post-provision.sh - Remove worker config from .env.template - Clean up commented-out worker blocks from main.bicep (~160 lines) - Revert text-embedding-3-large model deployment to active in ai-foundry.bicep - Clean up ailz-resources.env and post-deploy.sh references
The RBAC readiness probe previously tested GET /analyzers (read operation). On a brand-new AI Services resource, data-plane WRITE RBAC propagation takes 15-30 minutes — significantly longer than reads (~6 min). The read probe gave a false-positive, causing PATCH defaults to fail with HTTP 401 while write propagation was still in progress. Fix: replace the separate read-probe + PATCH-retry pattern with a single write-probe that uses the actual PATCH defaults call as the readiness test. Retries every 30s for up to 30 minutes (60 attempts), ensuring the script waits for write-level RBAC propagation. This resolves the consistent first-azd-up failure on fresh deployments.
The RBAC readiness probe previously tested GET /analyzers (read operation). On a brand-new AI Services resource, data-plane WRITE RBAC propagation takes 15-30 minutes — significantly longer than reads (~6 min). The read probe gave a false-positive, causing PATCH defaults to fail with HTTP 401 while write propagation was still in progress. Fix: replace the separate read-probe + PATCH-retry pattern with a single write-probe that uses the actual PATCH defaults call as the readiness test. Retries every 30s for up to 30 minutes (60 attempts), ensuring the script waits for write-level RBAC propagation. This resolves the consistent first-azd-up failure on fresh deployments.
Disable worker deployment
…om_content Extends the method in BaseExecutor to support [N] bracket notation within dot-separated field paths, enabling access to list elements in Content data. Example: 'result.contents[0].markdown' now correctly navigates content.data['result']['contents'][0]['markdown']. - Fully backward compatible with all existing dot-separated paths - Returns None safely for out-of-bounds, non-integer, or missing keys - Enables executors (azure_openai_agent_executor, content_classifier, etc.) to target specific array elements, reducing token usage and avoiding 429 rate limit errors when processing Azure Content Understanding results
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please review this pull request.